{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 运行日志\n",
    "  1. 训练日志\n",
    "    ![训练日志](https://gitee.com/zchsoft/learning-ai/raw/master/w11/train.png)\n",
    "  2. 验证日志\n",
    "    ![验证日志](https://gitee.com/zchsoft/learning-ai/raw/master/w11/eval.png)\n",
    "  3. 模型导出日志\n",
    "    ![模型导出日志](https://gitee.com/zchsoft/learning-ai/raw/master/w11/export_model.png)\n",
    "  4. 模型使用日志\n",
    "    1. 命令行方式\n",
    "      ![命令行方式](https://gitee.com/zchsoft/learning-ai/raw/master/w11/cmd_recog.png)\n",
    "    2. web server方式\n",
    "      ![命令行方式](https://gitee.com/zchsoft/learning-ai/raw/master/w11/server_recog.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 模型的训练流程\n",
    "  1. 准备数据集\n",
    "    1. 筛选有效数据\n",
    "    2. 构建适合所用网络的数据集\n",
    "  2. 修改网络，使分类器适合数据\n",
    "  3. 如果不是从头开始训练而是fine-tune，需要下载所使用网络的预训练模型\n",
    "  4. 训练\n",
    "  5. 训练过程中使用checkpoint进行模型验证，直到模型性能符合要求\n",
    "  6. 导出模型\n",
    "  7. 使用训练好的模型进行数据处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. tensorboard 分析\n",
    "  1. loss  \n",
    "    total_loss = clone_loss + regularization_loss  \n",
    "    clone_loss = softmax_cross_entropy_loss + aux_loss  \n",
    "    图中的clone_loss在最后还是有一定的抖动和下降趋势，应该还可以继续训练变的更好  \n",
    "    ![total_loss](https://gitee.com/zchsoft/learning-ai/raw/master/w11/total_loss.png)\n",
    "    ![regularization_loss](https://gitee.com/zchsoft/learning-ai/raw/master/w11/regularization_loss.png)\n",
    "    ![clone_loss](https://gitee.com/zchsoft/learning-ai/raw/master/w11/clone_loss.png)\n",
    "    ![losses](https://gitee.com/zchsoft/learning-ai/raw/master/w11/losses.png)\n",
    "    \n",
    "  2. global_step  \n",
    "    最开始时是使用的CPU训练，所以比较慢，后来改用GPU，训练速度有明显提升，大约是CPU速度的15倍：  \n",
    "    - CPU是i7-9750H，训练速是6.x秒/step，CPU占用率1000%左右（大约占了10个核心线程），虚拟内存很大，24G左右，实际使用RES不大；  \n",
    "    - GPU是GTX1660Ti，训练速度是0.4x秒/step, 基本占满了显存和GPU；  \n",
    "    ![global_step](https://gitee.com/zchsoft/learning-ai/raw/master/w11/global_step.png)\n",
    "    \n",
    "  3. learning_rate  \n",
    "    learning_rate从设置的初始值开始逐渐降低  \n",
    "    ![learning_rate](https://gitee.com/zchsoft/learning-ai/raw/master/w11/learning_rate.png)\n",
    "    \n",
    "  4. images\n",
    "    从图中可以看出前两个图是最后一张图中框出的部分\n",
    "    ![images](https://gitee.com/zchsoft/learning-ai/raw/master/w11/images.png)\n",
    "    \n",
    "  5. sparsity\n",
    "    参数矩阵的稀疏程度，稀疏度越高，模型越简单，模型的泛化能力越好  \n",
    "    ![sparsity_1.png](https://gitee.com/zchsoft/learning-ai/raw/master/w11/sparsity_1.png)\n",
    "    ![sparsity_2.png](https://gitee.com/zchsoft/learning-ai/raw/master/w11/sparsity_2.png)\n",
    "    \n",
    "  6. distributions\n",
    "    所有参数的分布变化趋近水平时，也可以说明训练过程基本结束了  \n",
    "    ![distributions_1.png](https://gitee.com/zchsoft/learning-ai/raw/master/w11/distributions_1.png)\n",
    "    ![distributions_33.png](https://gitee.com/zchsoft/learning-ai/raw/master/w11/distributions_33.png)\n",
    "  \n",
    "  7. histograms\n",
    "    ![histograms_1.png](https://gitee.com/zchsoft/learning-ai/raw/master/w11/histograms_1.png)\n",
    "    ![histograms_33.png](https://gitee.com/zchsoft/learning-ai/raw/master/w11/histograms_33.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
